fix(lookup_error): weak fuzzy hits no longer suppress semantic fallback#20
Merged
critesjosh merged 3 commits intomainfrom May 2, 2026
Merged
fix(lookup_error): weak fuzzy hits no longer suppress semantic fallback#20critesjosh merged 3 commits intomainfrom
critesjosh merged 3 commits intomainfrom
Conversation
Reported in v1.20.0 dogfood: `aztec_lookup_error("note already nullified")`
returned the unrelated catalog entry "Contract already initialized" with
a Jaccard word-overlap score of 54, and the semantic-documentation
fallback never fired because the early-return treated any catalog hit
— regardless of confidence — as authoritative.
Fix: introduce STRONG_MATCH_THRESHOLD = 70. Below that, the catalog
hits are kept in the result (so the formatter can still render them as
low-confidence cues), but the tool falls through to the semantic
fallback. The threshold aligns with the score system in
utils/error-lookup.ts:
100 exact-code / hex-signature → strong, always short-circuits
95 exact-pattern → strong
70-80 substring → strong (boundary at 70)
50-65 word-overlap (Jaccard) → weak, falls through
When weak hints exist alongside the semantic results, the message
field now names the situation ("No strong static match — N
low-confidence fuzzy hint(s) shown below. Showing relevant
documentation.") instead of pretending nothing matched.
Tests added (6):
- threshold boundary: score === 70 still short-circuits
- codeMatch alone (no catalog) still short-circuits
- regression: "note already nullified" with score-54 word-overlap →
semanticHealth='ok', semanticResults populated, weak hint preserved
in result.catalogMatches, message contains "low-confidence"
- multiple weak hints (max score 65) still fall through
- weak hints + no client → "skipped", message names the weak-hint
situation and points at API_KEY
- weak hints + semantic returning empty → "no_results", message
acknowledges both signals
All 247 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Formatter: when semantic results exist alongside weak-only catalog hits, render Documentation FIRST, then "## Lower-Confidence Catalog Hints" with an italicized note that the docs above are likely more authoritative. Prevents the LLM consumer from anchoring on a misleading top hit (the original "Contract already initialized" failure mode rendered the bogus entry under "## Known Errors" with full **bold name** + cause/fix before the actually-relevant docs). - category filter preserved: when the caller passes `category` and the catalog produced any in-category match (even a weak one), keep the pre-PR short-circuit. Falling through to a category-agnostic semantic search would surface out-of-scope docs and confuse a user who explicitly narrowed the request. - API_KEY guidance reworded from "An API_KEY would enable..." to "Set API_KEY... (get a free key by running /mcp-key in the Aztec/Noir Discord: https://discord.gg/xMud5StFyA)" — actionable next-step phrasing matches the rest of the project's wording. - 3 more tests covering the codex coverage gaps: * weak-only + version-mismatch: gate blocks semantic, weak hint preserved, message names both signals. * weak-only + allowVersionMismatch=true: gate skipped, semantic runs. * category filter + weak-only: short-circuits (does NOT fall through to category-agnostic semantic). 250 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex round-2 finding: the weak-only catalog note `_These are word-overlap fuzzy matches, not direct hits — the documentation results above are likely more authoritative._` rendered even when there were no semantic results above (no client, version mismatch, backend failed, semantic returned empty). Stale copy could mislead a user into thinking they should look up at a documentation section that doesn't exist. Now the "documentation above" phrasing is gated on `renderSemanticFirst` (which already encodes "semantic returned hits AND we reordered"). The other weak-only paths use neutral copy: `_Treat as low-confidence cues only._` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
critesjosh
force-pushed
the
feat/lookup-error-score-threshold
branch
from
38ca409 to
23d47f2
Compare
|
🎉 This PR is included in version 1.21.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reported in the v1.20.0 dogfood test:
aztec_lookup_error("note already nullified")returned the unrelated catalog entry "Contract already initialized" with a Jaccard word-overlap score of 54, and the semantic-documentation fallback never fired. The early-return inlookupAztecErrortreated anytotalMatches > 0as authoritative regardless of confidence, so a noise-floor fuzzy match shadowed the actually-useful answer.Fix
Introduce
STRONG_MATCH_THRESHOLD = 70. Catalog hits below the threshold no longer short-circuit semantic fallback — they're kept in the result so the formatter can still render them as low-confidence cues, but the tool now falls through to the semantic path.The threshold aligns with the existing score system in
src/utils/error-lookup.ts:exact-code/hex-signatureexact-patternsubstringword-overlap(Jaccard)A
codeMatch(ripgrep over cloned source) by itself still short-circuits — those are direct grep hits with no fuzziness.When weak hints exist alongside semantic results, the
messagefield names the situation explicitly ("No strong static match — N low-confidence fuzzy hint(s) shown below. Showing relevant documentation.") rather than pretending nothing matched.Tests added (6)
score === 70still short-circuitscodeMatchalone (no catalog) still short-circuits"note already nullified"with score-54 word-overlap →semanticHealth: "ok",semanticResultspopulated, weak hint preserved inresult.catalogMatches, message contains"low-confidence""skipped", message names the situation and points atAPI_KEY"no_results", message acknowledges both signalstsc --noEmitclean. All 247 tests pass (was 241).Test plan
npm run build(tsc)npx vitest runaztec_lookup_error("note already nullified")against v1.21+ and confirm semantic fallback returns documentation about note lifecycle / nullification.🤖 Generated with Claude Code